A Rule-Based Data Standardizer for Enterprise Data Bases

Whenever a database permits textual entry of information | for example when data is copied from a paper form | the database is likely to contain duplicates and inconsistencies. These duplicates must be removed and inconsistencies resolved in order to mine the data or to use the data for decision support. We term the domain-speci c solution to duplicate and inconsistency removal data standardization. In this paper, we describe a Name-Address Standardizer, one of a series of standardizers that have proven critical in creating a new enterprise-level database for the U.S. Customs Service. The standardizers were used to clean several legacy databases. These standardized databases were combined into a central database for which data is now standardized upon input. In practice, a standardizer uses techniques both from natural language analysis and from rule-based expert systems. As a result Prolog is highly suitable as a basis for standardizers. All Customs standardizers were written almost entirely in Prolog and constitute a large programming e ort: the Name-Address Standardizer contains about 100,000 lines of code, including generated parse tables and a fact base.